Human-assisted robotic picking apparatus

ABSTRACT

A robotic picking apparatus, system and method of use thereof. A system comprising: a plurality of robotic picking apparatuses configured to pick an item from a container and place the item in a different place, wherein each robotic picking apparatus is configured to operate in one of at least three operational modes: an autonomous mode, a human-assisted mode, and a remote-controlled mode, wherein in the autonomous mode, the robotic picking apparatus operates without human input, wherein in the human-assisted mode, an operation plan is determined using human input and the operation plan is executed automatically without additional input from a human, wherein in the remote-controlled mode, movements of the robotic picking apparatus are controlled directly by a remote human user. Human users located at one or more remote sites provide the human input.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional Application No.62/457,911 filed Feb. 12, 2017, entitled “REMOTE OPERATION OF ROBOTICPICKING STATIONS”, which is hereby incorporated by reference in itsentirety without giving rise to disavowment.

TECHNICAL FIELD

The present disclosure relates to robotic picking in general and tohuman-assisted robotic picking, in particular.

BACKGROUND

In a robotic warehouses, products are transferred to the packingstation, also referred to as a picking station. In the picking station,items are moved from container, also referred to as “tote” or “crate”,to container. For example, items may be moved from a box to a bag. Thepicking station may move items for different reasons or purposes, suchas to assemble an order for a customer, to perform shelf replenishment,QA, clearance of date-expired items, or other warehouse processes, orthe like.

In robotic warehouses, the picking stations may comprise of roboticpicking apparatus used to pick items from one container and move them toanother container. Various robotic picking apparatuses are known in theart, such as mimicking a human hand, an elephant's trunk, or the like.The robotic picking apparatus may be mounted on wheels or otherwise bemobile, thereby enabling the apparatus to move around and change itslocation, as well as to transport items that are carried thereby.

BRIEF SUMMARY

One exemplary embodiment of the disclosed subject matter is systemcomprising: a plurality of robotic picking apparatuses configured topick an item from a container and place the item in a different place,wherein each robotic picking apparatus is configured to operate in oneof at least three operational modes: an autonomous mode, ahuman-assisted mode, and a remote-controlled mode, wherein in theautonomous mode, the robotic picking apparatus is configured toautomatically determine an operation plan in order to pick the item fromthe container and place the item in the different place and execute theoperation plan without input from a human, wherein in the human-assistedmode, the operation plan is determined using input from a human and theoperation plan is executed automatically without additional input from ahuman, wherein in the remote-controlled mode, movements of the roboticpicking apparatus are controlled directly by a remote human user;wherein said plurality of robotic picking apparatuses are connectableover a network to a remote site, wherein in the remote site, human usersare capable of providing input to said plurality of robotic pickingapparatuses, wherein the input provided by the human users istransmitted to said plurality of robotic picking apparatuses over thenetwork, wherein the input provided by the human users comprises inputfor the human-assisted mode and commands for remote controlling therobotic picking apparatus in the remote-controlled mode.

Optionally, a controller is configured to select an operation mode of anapparatus of said plurality of robotic picking apparatuses from the atleast three operational modes, wherein said controller is configured togive preference to the autonomous mode over the human-assisted mode andthe remote-controlled mode, wherein said controller is configured togive preference to the human-assisted mode over the remote-controlledmode.

Optionally, a controller is configured to: obtain sensor input from aplurality of sensors at a vicinity of a robotic picking apparatus ofsaid plurality of robotic picking apparatuses; determine confidencemeasurement for the operation plan, wherein the operation plan isautomatically determined using the sensor input; in case the confidencemeasurement is above a threshold, provide the operation plan forexecution by the robotic picking apparatus in the autonomous mode; incase the confidence measurement is below the threshold, select one ofhuman-assisted mode or remote-controlled mode for the robotic pickingapparatus.

Optionally, said controller is configured to select human-assisted modeor remote-controlled mode based on a determination of whether humanprovided input can be used to determine the operation plan.

Optionally, said controller is configured to select between theautonomous mode, the human-assisted mode and the remote-controlled modebased on an estimated probability of success of operating in theautonomous mode, an estimated probability of success of operating in thehuman-assisted mode, and an estimated probability of success ofoperating in the remote-controlled mode.

Optionally, said controller is configured to select between theautonomous mode, the human-assisted mode and the remote-controlled modebased on an estimated cost of operating in the autonomous mode, anestimated cost of operating in the human-assisted mode, and an estimatedcost of operating in the remote-controlled mode.

Optionally, said controller is configured to utilize human input toimprove capabilities of the robotic picking apparatus in autonomousmode, whereby the autonomous mode is improved gradually based on humanprovided inputs of different types to allow picking of new items anditems in new scenarios.

Optionally, the remote site comprises a first remote site comprisinghuman-assistance stations and a second remote site comprising humantelecontrol stations; wherein said human-assistance stations areequipped with controls for a human to provide additional input to beused to determine operation plans for said plurality of robotic pickingapparatuses; wherein said human telecontrol stations are equipped withcontrols for a human to directly control movement of any of saidplurality of robotic picking apparatuses.

Optionally, the remote site is configured to be manned by a number ofhuman users for remotely assisting or controlling said plurality ofrobotic picking apparatuses, wherein a ratio between a total number ofhuman users in said system and a total number of robotic pickingapparatuses in said system is at least 1:10 or at least 1:100.

Optionally, in the human-assisted mode, the operation plan is determinedusing the input from a human that is used in the human-assisted mode,comprises at least one of: a user indication of the item in thecontainer, whereby assisting a computer vision algorithm to identify theitem; a user selection of the item between a plurality of items, wherebyassisting a selection algorithm to select the item to be picked; a userindication of a softness level of the item, whereby assisting indetermining amount of force to apply on the item during picking thereof;and a user indication of a location to put the item, whereby assisting aplacement algorithm to determine where to place the item.

Optionally, the container is a bulk container comprising a plurality ofitems of a same type, wherein the different place is a shippingcontainer comprising items based on an order, whereby said plurality ofrobotic picking apparatuses are configured to fulfill the order.

Optionally, the system further comprising: a picking station, whereinsaid picking station comprises at least one robotic picking apparatus;storage for storing containers; a plurality of transportation robotsconfigured to autonomously transport containers from said storage tosaid picking station.

Another exemplary embodiment of the disclosed subject matter is a methodcomprising: obtaining an instruction to pick an item from a container;first determining that an operation plan for a robotic picking apparatuscannot be devised autonomously; in response to said first determining,second determining that the operation plan cannot be devised based onadditional human provided input; and in response to said seconddetermining, receiving user instructions for remotely controlling therobotic picking apparatus, whereby the robotic picking apparatus isoperated in remote operation mode and controlled by a user; and movingthe robotic picking apparatus based on the user instructions.

Optionally, the method further comprising sending haptic feedback to theuser while the user is controlling the robotic picking apparatus.

Optionally, in response to said second determining, selecting a user outof a plurality of users to control the robotic picking apparatus,wherein said selecting is performed from a plurality of human userslocated remotely from the robotic picking apparatus.

Optionally, a ratio between a total number of human users that aretasked with assisting or controlling robotic picking apparatuses and atotal number of the robotic picking apparatuses is at least 1:100.

Optionally, said first determining comprises: calculating a firstconfidence measurement of the capability of autonomous determination ofthe operation plan, and comparing the first confidence measurement witha first threshold; and wherein said second determining comprises:calculating a second confidence measurement of the capability ofdetermination of the operation plan using additional human providedinput, and comparing the second confidence measurement with a secondthreshold.

Optionally, at least one of the first threshold and the second thresholdis modified based on availability of human users to provide input.

Optionally, the first threshold is modified based on availability ofmanned stations for remote assistance; and wherein the second thresholdis modified based on availability of manned stations for remote control.

Optionally, the method further comprising obtaining an order to fulfill;and fulfilling the order by instructing the robotic picking apparatus toplace items in a shipping container in accordance with the order,wherein said fulfilling comprises: the robotic picking apparatus pickingand placing at least one item in the shipping container in an autonomousmode and without user input; and the robotic picking apparatus pickingand placing at least one item in a human-assisted mode.

Optionally, the method further comprising obtaining an order to fulfill;and fulfilling the order by instructing the robotic picking apparatus toplace items in a shipping container in accordance with the order,wherein said fulfilling comprises: the robotic picking apparatus pickingand placing at least one item in the shipping container in aremote-controlled mode; and the robotic picking apparatus picking andplacing at least one item in the shipping container in an autonomousmode and without user input.

Yet another exemplary embodiment of the disclosed subject matter is amethod comprising: obtaining a instruction to pick an item from acontainer; computing an estimated cost of operating a robotic pickingapparatus in an autonomous mode, wherein the estimated cost of operatingthe robotic picking apparatus in the autonomous mode is based on aprobability of success of operating the robotic picking apparatus in theautonomous mode and estimated cost of damage to the item; computing anestimated cost of operating the robotic picking apparatus in ahuman-assisted mode, wherein the estimated cost of operating the roboticpicking apparatus in the human-assisted mode is based on a probabilityof success of operating the robotic picking apparatus in thehuman-assisted mode, the estimated cost of damage to the item and a costof labor in human-assisted model; computing an estimated cost ofoperating the robotic picking apparatus in a remote-controlled mode,wherein the estimated cost of operating the robotic picking apparatus inthe remote-controlled mode is based on a probability of success ofoperating the robotic picking apparatus in the remote-controlled mode,the estimated cost of damage to the item and a cost of labor inremote-controlled mode; selecting between the autonomous mode, thehuman-assisted mode and the remote-controlled mode based on theestimated costs; and operating the robotic picking apparatus to pick upthe item from the container in the selected mode.

Yet another exemplary embodiment of the disclosed subject matter is amethod comprising: obtaining an order of a customer to fulfill, whereinthe order comprises ordered items; instructing one or moretransportation robots to transport containers to a picking stationcomprising a robotic picking apparatus, wherein the containers comprisethe ordered items; picking, using the robotic picking apparatus, theordered items and placing the ordered items in a shipping container,wherein said picking comprises: picking at least one ordered item in anautonomous mode; picking one or more ordered items that cannot be pickedin the autonomous mode, wherein said picking the one or more itemscomprises: for each item of the one or more ordered items, determiningwhether to utilize a human-assisted mode or a remote-controlled mode forpicking the each item; obtaining human input based on the determinedmode; and picking the each item in the determined mode using the humaninput; and transporting the shipping container to the customer; wherebythe order is fulfilled by a robotic warehouse with limited humanassistance.

THE BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The present disclosed subject matter will be understood and appreciatedmore fully from the following detailed description taken in conjunctionwith the drawings in which corresponding or like numerals or charactersindicate corresponding or like components. Unless indicated otherwise,the drawings provide exemplary embodiments or aspects of the disclosureand do not limit the scope of the disclosure. In the drawings:

FIG. 1 shows an illustration of a system in accordance with someembodiments of the disclosed subject matter;

FIG. 2 shows a flowchart diagram of a method, in accordance with someexemplary embodiments of the disclosed subject matter;

FIG. 3A shows a flowchart diagram of a method, in accordance with someexemplary embodiments of the disclosed subject matter;

FIG. 3B shows a flowchart diagram of a method, in accordance with someexemplary embodiments of the disclosed subject matter; and

FIG. 4 shows a block diagram of a controller, in accordance with someexemplary embodiments of the disclosed subject matter.

DETAILED DESCRIPTION

One technical problem dealt with by the disclosed subject matter is toprovide a system of automated picking that requires limited humanintervention. In addition, it may be desired to provide a system wherehuman users are not located physically, so as to improve spaceutilization. In some exemplary embodiments, in many cases roboticwarehouses, such as for example robotic supermarkets, may be located inlocations where space is at a premium, and as a result improving spaceutilization may be a desired goal. Instead of providing a sufficientlylarge station where a person can work without being claustrophobic yethas convenient human-sized access, one, two or several robotic hands canmove products in and out of containers. This can significantly reducethe footprint of the packaging station and therefore of the roboticwarehouse.

However, robots are not as good as people at handling soft or unfamiliarproducts. Though mechanism to grip items exist, and such mechanisms mayemploy haptic feedback for sensing, the algorithms that manipulate thegripping mechanisms, are sometimes unable to handle items in certainconditions in an adequate manner. For example, the algorithm may applytoo much force and damage the product. As another example, the algorithmmay apply too little force and allow the product to slip and fall. Asyet another example, the algorithm may not be able to determine identifythe contour of the product and therefore may not be able to identify it.As yet another example, the algorithm may not be able to correctlychoose a gripping point in the product to grip, such as gripping acone-shaped product near its apex and therefore letting it slip due toits geometric shape. The above are exemplary potential shortcomings ofalgorithms in few scenarios. Additional shortcomings may be apparent toa person of ordinary skill in the art in view of the present disclosure.

One technical solution is to provide a system comprising a plurality ofrobotic picking apparatuses may be deployed in picking stations in arobotic warehouse. The robotic warehouse may utilize transportationrobots to carry and transport containers from one location to another.The robotic picking apparatuses may be configured to operate indifferent operational modes.

A first operational mode may be an autonomous mode. In the autonomousmode, the picking operation may be executed without any humanintervention or assistance. An operation plan may be determinedautomatically and executed by the robotic picking apparatus.

A second operational mode may be a human-assisted mode, where limitedhuman assistance may be provided. The human assistance may be additionalinput that is used for devising the operation plan for the roboticpicking apparatus. After such plan is determined, it may be executedautomatically without additional user input. As an example, the humanassistance may be identifying contour lines of the item to be picked,selecting an item from a bulk container to be picked, selecting a targetlocation where the item is to be placed, manually inputting acharacteristic of the product, such as softness level, viscosity level,wetness level, or the like (which may be inputted explicitly, such as byproviding a level in a scale, or indirectly such as indicating productshaving similar characteristic), or otherwise providing limited humanassistance that is useful for the determination of the operation planitself.

A third operational mode may be remote-controlled mode. Inremote-controlled mode, a user may remotely control the robotic pickingapparatus. The user may be directly control the movements of the roboticpicking apparatus. An electronical operation plan may not be devised,and instead the user may instruct the robotic picking apparatus how tomove during the picking operation. In some cases, the user may utilize adedicated device for controlling the movements of the robotic pickingapparatus and potentially receiving feedback, such as haptic feedbacktherefrom. In some exemplary embodiments, the user may utilize one ormore camera feeds to see the item to be picked and the state of therobotic picking apparatus, so as to be able to guide the robotic pickingapparatus and successfully pick up the item. In some exemplaryembodiments, the user may control a portion of the robotic pickingapparatus. For example, the robotic picking apparatus may comprise of arobotic arm and a gripping device. The robotic arm may be movedautonomously and the user may control the gripping device exclusively.

In some exemplary embodiments, a system in accordance with the disclosedsubject matter may comprise manned stations for remote assistance to aplurality robotic picking apparatuses. A relatively small number ofhuman users may be sufficient to assist a large number of roboticpicking apparatuses. For example, a ratio of 1:10, 1:100, 1:1000, oreven more, may be achieved. The robotic picking apparatuses may actautonomously most of the times, and when autonomous operation is notpossible or its adequacy is not ensured, human assistance may beprovided. Different degrees of assistance may be employed, withpreference of the human-assisted mode over the remote-controlled mode,as in the human-assisted mode a limited human attention may be requiredin comparison to the remote-controlled mode. Additionally oralternatively, in human-assisted mode, simpler and cheaper hardware maybe used, in comparison to the remote-controlled mode. In some cases, dueto the difference in costs of the hardware, there may be more mannedstations for remote assistance than manned stations for remote control.In some exemplary embodiments, there may be different users forassisting in different modes or different tasks of the same mode, so asto increase productivity in accordance with the division of laborprinciple. Additionally or alternatively, different human-input mayrequire use of different input devices, such as a touchscreen or otherpointing devices for human-assisted mode in providing assistance tocomputer vision algorithms, a textual input device for human-assistedmode in providing additional textual data to the algorithm responsiblefor determining pressure to apply on the item, a remote controlmimicking the shape of the robotic picking apparatus forremote-controlled mode, or the like. Additionally or alternatively,different human-input may be provided using different modalities, suchas gesture-based modality, pointing device modality, keyboard modality,or the like.

In some exemplary embodiments, a system may comprise of a plurality ofrobotic warehouses spread around the globe, each of which comprising alarge number of robotic picking apparatuses for performing warehouseoperations, such as order fulfillment, stock replenishment, or the like.The system may further comprise one or more human-centers, located in inareas where human labor is relatively inexpensive, such as India orChina. In the human-centers, a group of users may be prompted to assist,each time a different robotic picking apparatus, potentially usingdifferent methods. As a result, expensive labor and real-estate may bereplaced by cheap ones, while still operating the robotic warehousesaround the globe. Additionally or alternatively, as each robotic pickingapparatus may operate in autonomous mode most of the time, a number ofhumans that is smaller by one or more orders of magnitude than the totalnumber of robotic picking apparatuses employed around the globe, may besufficient to allow the plurality of robotic warehouses to functioncorrectly.

In some exemplary embodiments, human-assistance may be provided based oninformation provided to the user, as the user is not located at the samelocation as the robotic picking apparatus. As a result, the input whichthe user based her decision may be available to the system. Given theinput to the user, features may be extracted therefrom, and a classifiermay be trained with a pair of features and label. The label may be thehuman input given in view of the input. The classifier may provide aprediction of a correct answer and replace the human-assistancegradually. In some exemplary embodiments, the autonomous mode may employthe classifier and avoid relying on human input in similar cases in thefuture. In some exemplary embodiments, non-autonomous-mode may be usedin rare cases, such as when a new item is handled for the first time,when a new type of item is first encountered, when an uncommoncombination of items are encountered, or the like. In some exemplaryembodiments, the autonomous mode may be used in at least about 90% ofthe picking operations. In some cases, the autonomous mode may be usedin at least 99% of the picking operations. As a result, the amount ofhuman labor required may be limited and the system may scale withoutrequiring increasing manpower.

One technical effect of utilizing the disclosed subject matter is toprovide an efficient system for automated picking that can handle anypicking task. If a specific picking task cannot be completedautomatically, human assistance may be solicited, based on the level ofassistance required.

Another technical effect may be to provide for a scalable solution forunmanned robotic warehouses that utilize limited amount of premium realestate and premium manpower.

Yet another technical effect may be to improve the operability of therobotic picking apparatus. The disclosed subject matter may enable therobotic picking apparatus to operate in cases where it was previouslyimpossible. In some cases, based on human assistance, the controller ofthe robotic picking apparatus may learn how to handle in future cases,similar scenarios. The learning may be global learning that is usefulfor a variety of robotic warehouses. For example, in case of a newproduct that is launched, picking assignments of the new item may,initially, be handled using some human intervention. Based on the humanassistance in one robotic warehouse, all robotic warehouses may beenhanced to learn how to handle the new item. Similarly, in case where anew task has an uncommon condition, the human assistance may be leveledto not only solve the new task, but also assist in solving similar taskswith similar conditions to the uncommon condition, in any of the roboticwarehouses.

Yet another technical effect may be to improve fulfillment of onlineorders, such as using robotic warehouses that may be deployed withincities, and which may be capable of placing the items in a shippingcontainer without the presence of a human in the robotic warehouses forall types of orders and in all scenarios. The shipping container maythen be transported using drones, unmanned transportation devices, orother transportation method, reducing the overall fulfillment time fromthe placement of the order and until the ordered items reach thecustomer.

The disclosed subject matter may provide for one or more technicalimprovements over any pre-existing technique and any technique that haspreviously become routine or conventional in the art.

Additional technical problem, solution and effects may be apparent to aperson of ordinary skill in the art in view of the present disclosure.

Referring now to FIG. 1 showing an illustration of a system inaccordance with some embodiments of the disclosed subject matter.

FIG. 1 exemplifies a robotic warehouse having a Storage 100 for storingcontainers, such as Container 101. Storage 100 may comprise multipleshelving units for storing bulk containers. Transportation robots, suchas Lift Robot 103 and Floor Robot 102 may transport the containers,storing them in Storage 100, such as when new items arrive to therobotic warehouse, and retrieving them, such as for cleanup,replenishment, or order fulfilment. In some exemplary embodiments, therobotic warehouse may be similar to that disclosed in PCT ApplicationPCT/IB2016/053354, filed Jun. 8, 2016, entitled “ROBOTIC INVENTORYHANDLING”, published as WO 2016/199,033, which is hereby incorporated byreference in its entirety without giving rise to disavowment. Theillustrated embodiment shows a single robotic warehouse. However, thedisclosed subject matter is not limited to a single warehouse, and maybe applied for any number of warehouses.

The warehouse may comprise one or more Picking Stations 120. In theillustrated embodiment, three picking stations are shown. However, thedisclosed subject matter is not limited to such a number. A warehousemay comprise any number of picking stations, ranging from a singlepicking station to any conceivable number of picking stations.

Each Picking Station 120 may comprise a Robotic Picking Apparatus 130.In the illustrated embodiment, a one-to-one ratio of robotic pickingapparatuses to picking stations is shown. However, the disclosed subjectmatter is not limited to such a ratio, and in each station there may beany number of robotic picking apparatuses.

Robotic Picking Apparatus 130 may be configured to pick items fromcontainers (101) and place them in other locations, such as anothercontainer. The containers may be of the same size or of different sizes.For example, there may be several alternative container sizes for bulkcontainers, and different alternative container sizes for shippingcontainers. In some exemplary embodiments, Robotic Picking Apparatus 130may be controlled by a controller (not shown) that is configured toreceive input from sensors (not shown), such as video cameras, pressuresensors, Infrared (IR) cameras, or the like, and control the operationof Robotic Picking Apparatus 130. In most cases, controller may employan algorithm to formulate an operation plan. The plan may comprise a setof movements of Robotic Picking Apparatus 130 to be performed in orderto perform the picking operation, to place the item in the new location,or the like. In some exemplary embodiments, Robotic Picking Apparatus130 may be mobile, such as may be mounted on a wheeled platform, and theoperation plan may comprise planned route for the mobile Robotic PickingApparatus 130. Additionally or alternatively, the operation plan mayindicate forces to be applied, such as amount of pressure to be usedwhen gripping the item.

In some exemplary embodiments, an operation plan cannot be devisedautomatically. In some cases, additional human input, may be sufficientto devise the plan. For example, the plan may not be devised because theitem cannot be located by the computer vision algorithm in picture ofthe container. In such a case, human assistance stations in a remotesite may be contacted via a Network (140). User 150 in the humanassistance station may be prompted to provide human assistance, such asindicating, using a touchscreen of a Tablet 155A or using a point deviceof a Desktop Computer 155B, where the item is located. The picture or avideo feed from the sensors may be shown to User 150 on the screen, andthe user may point on the item, indicate its contour and visibleborders, or the like. Additionally or alternatively, the humanassistance may include pointing to a location where to place the item,such as the target location within a half-packed container, indicatingabove which items the picked item should be placed. The placementdecision may be based on considerations such as weight, firmness,boxing, sizes of the items, or the like. Additionally or alternatively,a selection algorithm for selecting which item to pick up out of aplurality of identical items (e.g., in a bulk container) may be unableto make a selection. Although the items are identified by the computervision algorithm, the selection of an item to pick up may be non-trivialdue to the relationship between the items and potential implications ofpicking up each items, such as shifting the location of other items. Insome cases, the selection algorithm may require assistance in case ofnon-identical items of the same type. For example, in case of a bulkcontainer comprising tomatoes, each tomato may unique, and the selectionalgorithm may not be confident in a selection of one tomato over theothers. The user may select the item, such as by pointing to theselected on a screen. Additionally or alternatively, a movement route ofthe robotic picking apparatus may not be determined automatically. Theuser may assist by selecting a target destination, changing suggestedlegs in the route, setting minimal and maximal speed limits, or thelike. Additionally or alternately, softness level of the item or othercharacteristics thereof, may be required for the algorithm to devise theoperation plan. If the algorithm cannot assess in sufficient confidence(e.g., confidence level above a threshold) the characteristic, the usermay be prompted to provide the response. In some exemplary embodiments,the threshold may depend on the expected outcome of using a wrongcharacteristic. For example, if the characteristic is softness, then thedamage may be expected to be greater than if the characteristic iscolor, as picking the item automatically when mistaking the softnesscharacteristic may cause the item to be damaged in the process withhigher likelihood than with the color characteristic. As anotherexample, if the characteristic is softness and the estimation is thatthe item is soft, the threshold may be lower than if the estimation isthat the item is firm, as if the latter is erroneous, the expecteddamage to the item is higher. The user may be shown the picture of theitem, estimated characteristic, confidence level, and the like. The usermay then confirm the estimation or modify it. The user may utilize anobjective scale to indicate the characteristic. As another option, theuser may select another product, indicating that the current item hassimilar characteristic to the selected item. In some cases, the user mayindirectly indicate the characteristic by indicating the relationshipbetween the item and other items. For example, the user may be askedabout the softness of an orange. The user may be prompted with a pictureof a tomato, and asked whether the item of interest, the orange, issofter or harder than the tomato. After indicating that orange is harderthan tomato, the user may be asked to indicate relationship with a tincan, and may indicate that the orange is softer than the tin can. Insuch a manner, using other objects, the user may indicate the softnesslevel in a relatively accurate manner, and without using an objectivescale.

In some exemplary embodiments, in case the operation plan cannot bedevised even after receiving the human input, or if it is determineda-priori that additional input would not be sufficient to devise theplan, remote-control mode may be employed. Human telecontrol stations inthe remote site may be contacted via Network 140. In some exemplaryembodiments, User 160 may utilize the same devices as User 150 tocontrol the Robotic Picking Apparatus 130. In such a case, the same usermay provide assistance in human-assisted mode and tele-operate theRobotic Picking Apparatus 130, in remote-controlled mode. In someexemplary embodiments, users providing input in human-assisted mode maybe different than users performing teleoperation of the pickingapparatus. This could be due to an improved efficiency and utilization,by having each user specialize in performing a specific task.Additionally or alternatively, this may be the case if different toolsare used to provide input in the different modes. For example, User 160may utilize dedicated devices to control the Robotic Picking Apparatus130, such as a glove (165A) for precisely controlling Gripping Device132, or a remote control (165B). Additionally or alternatively, humantelecontrol stations may employ different modalities than those employedby human assistance stations.

In some exemplary embodiments, the determination whether to contact ahuman assistance station, human telecontrol station, or the like, may bebased on the availability of human users. If all users are currentlybusy and unavailable, the minimal confidence threshold of an estimation,such as estimation of the computer vision algorithm, the selectionalgorithm, or the like, may be reduced, thereby relying on estimationsthat are less reliable. However, it may be preferred to perform theaction or at least attempt performing it, without waiting for a humanuser to become available, and therefore reduce the overall time requiredto perform the task at hand. Similarly, if no users are available forteleoperation, the minimal confidence level in the adequacy of theoperation plan may be reduce, thereby attempting to autonomously performplans that have higher probability to fail than what is usuallyacceptable. In some exemplary embodiments, in case all human assistancestations are busy but some human telecontrol stations are not, theRobotic Picking Apparatus 130 may be operated in remote-controlled mode,even though it could have been operated in human-assisted mode. It isnoted that such an embodiment may improve efficiency of the system,improve utilization of the human manpower resource, and may reduce theoverall time required to perform all tasks.

Referring now to FIG. 2 showing a flowchart diagram of a method, inaccordance with some exemplary embodiments of the disclosed subjectmatter.

On Step 200, an order is obtained. The order may define a set of ordereditems to be shipped to a customer making the order. It is noted thatillustrated figure relates to fulfilling a customer order, however,similar methods may be employed to perform other warehouse processesthat involve the moving of items from or into containers.

On Step 210, an item to be placed in the shipping container is selected.In some exemplary embodiments, the order may describe an order ofplacing the items in the containers, such as which items are placed inthe bottom of the container and which are placed above. Such placementorder may impose an order of selection of the items. In some exemplaryembodiments, the placement order may be automatically determined, suchas disclosed in an international patent application being concurrentlyfiled, entitled PACKING AND DELIVERY OF GOODS AMONG MULTIPLE STATIONS,which claims the benefit of US U.S. Provisional Application No.62/457,912 filed Feb. 12, 2017, entitled “DIVIDING ITEMS ORDERED AMONGMULTIPLE PACKING STATIONS”, both of which are hereby incorporated byreference in their entirety without giving rise to disavowment.

On Step 220, a bulk container comprising the selected item may betransported to a picking station. The bulk container may be retrieved bya lift robot from a shelving unit of the storage, and may be mounted ona floor robot to be transported to the picking station. It is noted thatother combination of robots, including a single multi-purpose robot, maybe employed to transport the bulk container from storage to the pickingstation. The bulk container may comprise a plurality of items of thetype of the selected item. For example, a plurality of oranges, if theselected item is an orange; a plurality of beer bottles of a brand, ifthe selected item is a beer bottle of the brand; or the like. In somecases, the bulk container may comprise a plurality of items of differenttypes and may not necessarily comprise only items of the selected type.

In some exemplary embodiments, the picking station may be unmanned andmay comprise one or more robotic picking apparatuses. In some exemplaryembodiments, the picking station may also comprise a shipping container,such as a container that will be transported, manually or using one ormore robots, to the customer. In some exemplary embodiments, if at thebeginning of processing of an order, there is no shipping container atthe picking station handling the order, an empty shipping container maybe transported thereto by a robot, such as by extracting the shippingcontainer from the storage.

On Step 230, the selected item is picked up from the bulk container andplaced in the shipping container, by the robotic picking apparatus. Insome exemplary embodiments, the picking and placing may be performedfully autonomously and without user intervention. In some exemplaryembodiments, and optionally, only at relatively rare cases, userassistance may be solicited. Remote users located at one or more remotesites may be prompted to assist the robotic picking apparatus. In somecases, different types of assistances and level of engagement may beavailable. In human-assisted mode, the user may provide additional inputon top of sensor input that can be used to automatically determine anoperation plan for the robotic picking apparatus. If such input is notpractical, such as because it would not be sufficient to determine aplan, or because there are no available users that can provide suchinput, input of a different form may be solicited. In some exemplaryembodiments, users utilizing remote-control devices may be prompted toremotely operate the robotic picking apparatus in remote-controlledmode.

In some exemplary embodiments, Steps 210-230 may be performediteratively until all items of the order are placed in the shippingcontainer.

On Step 240, the shipping container may be transported to the customer,thereby fulfilling the order. In some exemplary embodiments, thetransportation of the shipping container may be performed by unmanneddrones and robots, such as the floor robots, airborne drones, or thelike. Additionally or alternatively, the transportation may utilizetransportation services, such as provided by UPS™, DHL™, or the like,and hence may or may not be performed, at least in part, by humans.

Reference is now made to FIG. 3A, which shows a flowchart diagram of amethod, in accordance with some exemplary embodiments of the disclosedsubject matter.

On Step 300, an operation plan for the robotic picking apparatus isdetermined. In some cases, a software component may be configured toattempt to formulate the operation plan based on inputs available fromsensors at the vicinity of the robotic picking apparatus, such ascamera, microphone, pressure sensors, or the like. In some cases, theplan may require applying a series of different algorithms, such ascomputer vision algorithms for identifying the item in the image; aselection algorithm for selecting which of a plurality of items of thesame type to pick up (e.g., based on the heap configuration and onpotential movements in response to picking thereof); an algorithm todetermine amount of force to apply on the picked-up items (e.g., basedon their perceived firmness); a placement algorithm to determine a placeto place the item (e.g., a location within a target container, based onthe current items placed therein and based on additional items to beplaced after the current item); a route planning algorithm for planninga route for the robotic picking apparatus, such as to move to a locationto pick up the item, to move to a location to drop the item, or thelike; and so forth.

In some exemplary embodiments, an attempt may be made to formulate anoperation plan. The attempt may be successful and a suggested operationplan may be devised. Additionally or alternatively, the attempt may failand no suggested plan may be available.

On Step 310, a determination whether an adequate plan was created may bemade. In some exemplary embodiments, if a suggested operation plan wasdetermined, a confidence level in its successful application may becalculated. The confidence level may be based, for example, on a historyof successful or unsuccessful execution of operation plans. For example,if the success rate of picking up a tomato is below about 80%, it may bedetermined that the operation plan is inadequate. Additionally oralternatively, the success rate of a suggested trajectory may becomputed and used (e.g., 10% that a tomato that is picked up in anaccess trajectory of a generally diagonal vector would be smashed).

In some exemplary embodiments, the confidence threshold may be differentfor different items or orders. For example, confidence threshold forfragile items may be higher than for robust items. As another example,confidence threshold for expensive items may be higher than of cheapitems. As yet another example, confidence threshold of items with lowinventory availability may be higher than confidence threshold for itemsthat are full in stock. Such different thresholds may be used to providefor improved resource allocation of human users, and reduced estimatedcost of damage to items when automatically executing operational plans.The estimated cost of the damage may include the cost of the damagedgoods itself, the probability of damaging the item, the probability ofrendering the order unfulfillable until inventory is replenished, costand probability of cleanup, cost and estimated probability of wastedtime, estimated cost of adversely affecting the item without damaging it(e.g., wrinkling or otherwise deforming the package without damaging itscontent), or the like.

In some exemplary embodiments, adequacy may be computed by taking intoaccount various parameters, such as expected probability of success ofthe plan, expected probability of success of using a human user, cost oflabor, cost of mistake in execution, probability in mistake inexecution, or the like. In some exemplary embodiments, if the exceptedprobability of success of the plan is at least as high as an exceptedprobability of success of a human remote-controlling the device, theplan may be considered adequate. If, on the other hand, there is anadvantage to using a human (e.g., higher probability of success to humantele-control or to human assistance), then the determination may be madeby computing a net value of utilizing human assistance. In someexemplary embodiments, the net value of human-assistance may be computedby computing estimated cost of execution using human assistance andsubtracting cost of automated execution: NV_(mode)=EC_(mode)−EC_(auto),where NV_(mode) is the net value of using mode (e.g., human-assistedmode, remote-controlled mode) instead of using the autonomous mode,EC_(mode) is estimated cost of using mode (e.g., autonomous mode,human-assisted mode, remote-controlled mode, or the like).EC_(mode)=(1−p_(mode)). D_(i)+L_(mode), where p_(mode) is theprobability of failure causing damage when using mode, D_(i) is theexpected damage to item i in case of a failure, L_(mode) is the cost oflabor in the mode (0 is in case of autonomous mode). In some cases, adifferent types of damages may be referred to and each may have adifferent probability of occurring (e.g., EC_(mode)=Σ_(j) (1−p_(mode)^(j))·D_(i) ^(j)+L_(mode), where p_(mode) ^(j) is the probability ofdamage of type j when operating in mode, and D_(i) ^(j) is the expecteddamage to item i when damage of type j occurs). In some exemplaryembodiments, the selection of the human-assisted mode may be based onthe NV_(human-assisted) being positive. Additionally or alternatively,EC_(auto), EC_(human-assisted) and EC_(remote-controlled) may becompared and the mode may be selected based on the mode having theminimized associated estimated cost.

It is noted that in some cases, an operation plan may appear to beadequate and upon execution, such plan may fail. In such a case, Step310 may be re-performed and in such a case, the determination may bethat there is no adequate plan at hand.

Additionally or alternatively, instead of attempting to devise a planautomatically and determine that such attempt was unsuccessful, it maybe a-priori estimated, given the sensor inputs available and the task athand, whether the system is capable to autonomously determine theoperation plan. Such estimation may be accompanied by a confidence levelthat may compared with a threshold confidence. If the system estimatesthat a plan can be devised autonomously, with a confidence level higherthan the minimal threshold (e.g., above 80% confidence), than the planmay be autonomously determined and utilized.

It will be determined that in the present application, a confidencemeasurement of a capability of autonomous determination of the operationplan relates to a confidence measurement of an estimation prior toattempt to create a plan and also to a confidence measurement in theadequacy of a determined plan posteriori.

On Step 320, in case an adequate operation plan was automaticallydevices, the plan is executed by the robotic picking apparatus inautonomous mode. Otherwise, the task at hand may not be performed in theautonomous mode and Step 330 may be performed.

On Step 330, it may be determined whether partial assistance issufficient to overcome the inadequacy of the operation plan.

In some exemplary embodiments, the failure to formulate an operationplan may be caused due to specific failure of a specific algorithm orlack of confidence in the products of such algorithm. In such a case,human input can be solicited to overcome the deficiencies of suchalgorithm and generate the operation plan using the human input. Forexample, a human user may be contacted and shown the image of the bulkcontainer. The user may then identify contours of the item to be picked.As another example, the user may be shown the image of the bulkcontainer with highlighted contour lines of each identified item, andthe user may select an item to be picked up out of the plurality ofidentified items (or indicate that the item to pick up was notautomatically identified and manually identify it). As yet anotherexample, the user may indicate one or more gripping points in the item,where the gripping mechanism is to apply pressure in order to pick upthe item. For example, the user may indicate two gripping points inopposite locations in the item, in order for a gripping mechanismemulating an elephant trunk to be able to pick up the item. Additionallyor alternatively, the user may indicate a location where a suction cupcan be placed in order to apply vacuum to pick up the item. As yetanother example of human assistance, an image showing the inside of theshipping container may be presented to the user, who may indicate alocation to place the item after picking it up. As yet another example,the user may be prompted to indicate a property of the item that isselected to be picked. For example, firmness of the item may beindicated. Additionally or alternatively, a range of potential firmnessproperties, such as in case of a tomato which may be hard or soft,depending on its ripeness. Additionally or alternatively, the user mayindicate the property after receiving feedback. For example, the roboticpicking apparatus may attempt picking up the item, and tactile feedbackmay be provided to the user to verify or modify her initial estimation.

In some exemplary embodiments, human assistance may be solicited whenthe algorithm provides an estimated result, however, the confidencelevel in the estimation is below a threshold. In such a case, the usermay be solicited to verify or nullify the estimated result that isautomatically determined. For example, the user may be prompted toverify that the estimated contour of the item was indeed correctlyidentified.

In some exemplary embodiments, such human assistance is local andrelatively minimal to specific problems that the algorithms encountered.If given such additional input, automated process may determine andexecute a plan, Step 350 may be performed.

In some exemplary embodiments, the determination whether partialassistance is sufficient may be performed using a prediction model, aclassifier, or the like. Confidence measurement of the prediction may beused so as to ensure relying on the prediction only given a strongenough confidence in the prediction.

On Step 350, a human user is prompted to provide the human assistance.In some exemplary embodiments, the user may be located in a remote site,such as in a service center located in a different country. The user maybe in a manned station for remote assistance.

In some exemplary embodiments, the user may be selected from a pluralityof users that man such stations. In some cases, the user may be selectedbased on her availability, historic performance metrics (e.g., how fastand correct the user solves problems similar to the problem at hand), orthe like.

The user may provide her assistance, and based thereon, a plan to pickup the item may be determined (Steps 300-310) and potentially usedautonomously (320). In some cases, human-assistance may be solicitedonce more (330-350). If in the end, an operation plan is determinedusing the sensor input combined with additional human input, andexecuted without user control, then the robotic picking apparatus may bereferred to as operating in human-assisted mode.

If, on the other hand, it is determined that partial assistance isinsufficient to create an adequate plan, then the robotic pickingapparatus may be operated in remote-controlled mode (370).

On Step 370, a user may be prompted to telecontrol the robotic pickingapparatus. In some exemplary embodiments, the user may be in a mannedstation for remote control, at a remote site. The user may be selectedbased on her availability, historic performance in similar tasks, or thelike. The user may utilize dedicated remote control apparatuses forcontrolling the robotic picking apparatus. Inputs from the sensors ofthe robotic picking apparatus may be provided to the user, and the usermay, in response, provide control instructions to the robotic pickingapparatus. In some exemplary embodiments, haptic feedback may beprovided to the user to allow her to determine the correct amount ofpressure to apply on the item.

In some exemplary embodiments, machine learning may be employed toimprove the performance of the system over time. In some exemplaryembodiments, control instructions in remote-controlled mode (375) may beused to improve the capabilities of the planning module. In someexemplary embodiments, the control instructions may be viewed as anoperation plan that could have been executed automatically. Once thetask is completed, such operation plan may be used to train the planningmodule to improve it so that if a similar scenario is encountered again,the planning module may be capable of providing an adequate operationplan without user intervention. Additionally or alternatively, input ofthe human-assisted mode may be utilized to improve the algorithms usedby the planning module. For example, computer vision algorithms may betrained using the image shown to the user and the human input providedby the user, so that in future cases, if similar cases are encounteredagain, the system may solve the problem in a high enough confidencewithout human assistance.

In some exemplary embodiments, employing machine learning techniques mayimprove the system gradually to allow picking of new items and items innew scenarios. In some exemplary embodiments, the learning may beperformed based on human input provided for one warehouse and applied toa second warehouse. For example, if a new item becomes available atseveral warehouses, it may be sufficient for a human user to assist thesystem once in identifying the item with respect to one warehouse, andafter such assistance, all the warehouses may be capable ofautomatically handling such item without human assistance.

In some exemplary embodiments, the system may be biased towardsoperating in autonomous mode, and if such operation is not possibletowards using human-assisted mode rather than remote-controlled mode. Insome cases, availability of human users to provide assistance may affectthe biasing scheme of the system. In some exemplary embodiments, in caseof low availability of human users to assist (in either of thehuman-associated modes), borderline operational plans may be consideredas adequate. In some exemplary embodiments, in case there are noavailable human users, the confidence threshold may be reduced (305),such that operational plans with a confidence measurement that is belowthe regular threshold but above the modified threshold may be consideredas adequate. For example, instead of requiring a confidence if 80% ormore, a plan with 75% confidence may be considered adequate in lowavailability cases. In some exemplary embodiments, in case there is highavailability of human users that can provide input for human-assistedmode, but low availability of users that can tele-operate the roboticpicking apparatus, confidence thresholds for determining that partialassistance is sufficient may be modified, to enable use ofhuman-assisted mode in cases where under regular conditions,remote-controlled mode would be used. Additionally or alternatively, incase partial assistance is not available (340), remote-controlled modemay be used instead thereof. In some exemplary embodiments, suchmodifications increase overall utility of the system as a whole. Insteadof relying on specific form of assistance and waiting until relevantresources become available, alternative methods may be employed. In suchcases, less-optimal plans may be executed and goods may be damages inthe process. As such, the modification of thresholds may be differentfor different items, such as depending on their cost or value, dependingon their estimated damage when dropped, or the like. For example, theconfidence threshold for a fragile artifact may not be modified whilethe confidence measurement for robust items may be modified. As anotherexample, the confidence threshold of an expensive ROLEX™ watch may notbe modified or may be modified using a different modification scale thanthe modification of a less expensive unbranded watch, so as to result ina higher confidence measurement for the branded watch.

Reference is now made to FIG. 3B, which shows a flowchart diagram of amethod, in accordance with some exemplary embodiments of the disclosedsubject matter. FIG. 3B shows a method similar to that of FIG. 3A.However, instead of determining a plan in a step-by-step manner, byplanning each item pick up task individually, in the illustrated methodof FIG. 3B, a plan to fulfill an entire order is devised (301).

Referring now to FIG. 4 showing a block diagram of a controller, inaccordance with some exemplary embodiments of the disclosed subjectmatter. A Controller 400 may be configured to support parallel userinteraction with a real world physical system and a digitalrepresentation thereof, in accordance with the disclosed subject matter.

In some exemplary embodiments, Controller 400 may be deployed within therobotic warehouse itself, different Controllers 400 may be deployed,each of which controlling a different set of one or more robotic pickingapparatuses. Additionally or alternatively, Controller 400 may bedeployed in a remote location, such as in a cloud server, connectable tothe robotic picking apparatuses via a computerized network.

In some exemplary embodiments, Controller 400 may comprise a Processor402. Processor 402 may be a Central Processing Unit (CPU), amicroprocessor, an electronic circuit, an Integrated Circuit (IC) or thelike. Processor 402 may be utilized to perform computations required byController 400 or any of it subcomponents.

In some exemplary embodiments of the disclosed subject matter,Controller 400 may comprise an Input/Output (I/O) module 405. I/O Module405 may be utilized to provide an output to and receive input from auser, such as, for example obtain information from Sensors 450 at thevicinity of the robotic picking apparatus, to communicate with, senddata to and receive instructions from manned stations, such asManned-Stations for Human-Assistance 460 and Manned-Stations for RemoteControl 470, or the like.

In some exemplary embodiments, Controller 400 may comprise Memory 407.Memory 407 may be a hard disk drive, a Flash disk, a Random AccessMemory (RAM), a memory chip, or the like. In some exemplary embodiments,Memory 407 may retain program code operative to cause Processor 402 toperform acts associated with any of the subcomponents of Controller 400.Memory 407 may retain the order, including the ordered items, the orderof picking up items, or the like. Memory 407 may retain the operationalplan. Additionally or alternatively, Memory 407 may retain history ofoperational plans, outcomes thereof, training dataset obtained based onuser input, such as verification of automated determination, userinstructions, or the like.

Planning Module 410 may be configured to plan an operational plan forpicking up and item and placing it in a different location. PlanningModule 410 may be configured to employ one or more algorithms, such asselection algorithm, computer vision algorithm, planning algorithm,route planning algorithm, or the like. Planning Module 410 may beconfigured to utilize inputs from Sensors 450 to formulate theoperational plan. Additionally or alternatively, Planning Module 410 mayoutput a confidence measurement in the successfulness of the determinedplan. Additionally or alternatively, Planning Module 410 may be utilizeadditional input from a human user, such as provided by Manned Stationsfor Human-Assistance 460, to determine the plan. The Planning Module 410may utilize the human input to override, modify, or otherwise change anautomated determination by Planning Module 410. Additionally oralternatively, the human input may be used to provide a parameter wherenone of the algorithms are capable of providing it automatically, suchas when the item is not identified by any of the employed computervision algorithms. Additionally or alternatively, the human input may beused to verify borderline automated determinations.

Confidence Determinator 420 may be configured to set a confidencethreshold. The confidence threshold may be used for determining whethera plan is adequate. The confidence threshold may be used for determiningwhether a determination that partial assistance can be used is reliable.In some exemplary embodiments, different confidence measurements may beused for different purposes. Confidence Determinator 420 may beconfigured to set different confidence measurements for different items,such as depending on their cost, stock availability, robustness, or thelike. Confidence Determinator 420 may be configured to modify theconfidence based on availability of human users. In some exemplaryembodiments, availability of Manned Stations for Human-Assistance 460may affect the confidence threshold of when remote assistance issolicited. Additionally or alternatively, availability of MannedStations for Human-Assistance 460 may affect confidence threshold ofwhen a plan is considered adequate. Additionally or alternatively,availability of Manned Stations for Remote Control 470 may affectconfidence threshold of when a plan is considered adequate, of whenpartial remote assistance is solicited, or the like.

Mode Selector 430 may be configured to select a mode of operation for arobotic picking apparatus. Mode Selector 430 may be configured to selectfrom one of at least three modes of operation: an autonomous mode, ahuman-assisted mode, and a remote-controlled mode. In some exemplaryembodiments, Mode Selector 430 is configured to bias the selectiontowards autonomous mode. Additionally or alternatively, Mode Selector430 may be configured to bias the selection of human-assisted mode overremote-controlled mode. In some exemplary embodiments, Mode Selector 430may be configured to make the mode selection based on availability ofmanned stations that can provide assistance or instructions (460, 470).Mode Selector 430 may be configured to determine the mode based onexpected cost of failure of the operational model, such as probabilityof damage, cost of item, probability of lack of replacement item, or thelike.

Machine Learning Module 440 may be employed to gradually improveperformance of the robotic picking apparatuses. Machine Learning Module440 may utilize human input obtained from manned stations (460, 470) toimprove automated determination of Planning Module 410. In someexemplary embodiments, a feature vector may be extracted from the inputof the Sensors (450) that is relayed to the manned stations (460, 470).A label for the feature vector may be determined based on the humaninput, such as a solution to a particular problem posed in thehuman-assisted mode, a set of control instructions provided in theremote-controlled mode, or the like. The pair of feature vectors andassociated label may be provided for training a prediction model andimproving its performance. Additionally or alternatively, the pair maybe used to verify or refute automatically determined values to provide afeedback on the success rate of each algorithm. Other machine learningtechniques may be employed to utilize the human-verified data.

It is noted that in some embodiments, machine learning may be employedin a center location, such as in a cloud server. The cloud server maytrain its predictive models and update the predictive models in variousControllers 400.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or act for performing the function incombination with other claimed elements as specifically claimed. Thedescription of the present invention has been presented for purposes ofillustration and description, but is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the invention. Theembodiment was chosen and described in order to best explain theprinciples of the invention and the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated.

1. A system comprising: a plurality of robotic picking apparatusesconfigured to pick an item from a container and place the item in adifferent place, wherein each robotic picking apparatus is configured tooperate in one of at least three operational modes: an autonomous mode,a human-assisted mode, and a remote-controlled mode, wherein in theautonomous mode, the robotic picking apparatus is configured toautomatically determine an operation plan in order to pick the item fromthe container and place the item in the different place and execute theoperation plan without input from a human, Wherein in the human-assistedmode, the operation plan is determined using input from a human and theoperation plan is executed automatically without additional input from ahuman, wherein in the remote-controlled mode, movements of the roboticpicking apparatus are controlled directly by a remote human user;wherein said plurality of robotic picking apparatuses are connectableover a network to a remote site, wherein in the remote site, human usersare capable of providing input to said plurality of robotic pickingapparatuses, wherein the input provided by the human users istransmitted to said plurality of robotic picking apparatuses over thenetwork, wherein the input provided by the human users comprises inputfor the human-assisted mode and commands for remote controlling therobotic picking apparatus in the remote-controlled mode.
 2. The systemof claim 1, wherein a controller is configured to select an operationmode of an apparatus of said plurality of robotic picking apparatusesfrom the at least three operational modes, wherein said controller isconfigured to give preference to the autonomous mode over thehuman-assisted mode and the remote-controlled mode, wherein saidcontroller is configured to give preference to the human-assisted modeover the remote-controlled mode.
 3. The system of claim 1, wherein acontroller is configured to: obtain sensor input from a plurality ofsensors at a vicinity of a robotic picking apparatus of said pluralityof robotic picking apparatuses; determine confidence measurement for theoperation plan, wherein the operation plan is automatically determinedusing the sensor input; in case the confidence measurement is above athreshold, provide the operation plan for execution by the roboticpicking apparatus in the autonomous mode; in case the confidencemeasurement is below the threshold, select one of human-assisted mode orremote-controlled mode for the robotic picking apparatus.
 4. The systemof claim 3, wherein said controller is configured to selecthuman-assisted mode or remote-controlled mode based on a determinationof whether human provided input can be used to determine the operationplan.
 5. The system of claim 3, wherein said controller is configured toselect between the autonomous mode, the human-assisted mode and theremote-controlled mode based on an estimated probability of success ofoperating in the autonomous mode, an estimated probability of success ofoperating in the human-assisted mode, and an estimated probability ofsuccess of operating in the remote-controlled mode.
 6. The system ofclaim 3, wherein said controller is configured to select between theautonomous mode, the human-assisted mode and the remote-controlled modebased on an estimated cost of operating in the autonomous mode, anestimated cost of operating in the human-assisted mode, and an estimatedcost of operating in the remote-controlled mode.
 7. The system of claim3, wherein said controller is configured to utilize human input toimprove capabilities of the robotic picking apparatus in autonomousmode, whereby the autonomous mode is improved gradually based on humanprovided inputs of different types to allow picking of new items anditems in new scenarios.
 8. (canceled)
 9. The system of claim 1, whereinthe remote site is configured to be manned by a number of human usersfor remotely assisting or controlling said plurality of robotic pickingapparatuses, wherein a ratio between a total number of human users insaid system and a total number of robotic picking apparatuses in saidsystem is at least 1:10.
 10. (canceled)
 11. The system of claim 1,wherein in the human-assisted mode, the operation plan is determinedusing the input from a human that is used in the human-assisted mode,comprises at least one of: a user indication of the item in thecontainer, whereby assisting a computer vision algorithm to identify theitem; a user selection of the item between a plurality of items, wherebyassisting a selection algorithm to select the item to be picked; a userindication of a softness level of the item, whereby assisting indetermining amount of force to apply on the item during picking thereof;and a user indication of a location to put the item, whereby assisting aplacement algorithm to determine where to place the item.
 12. The systemof claim 1, wherein the container is a bulk container comprising aplurality of items of a same type, wherein the different place is ashipping container comprising items based on an order, whereby saidplurality of robotic picking apparatuses are configured to fulfill theorder, wherein the system further comprises: a picking station, whereinsaid picking station comprises at least one robotic picking apparatus;storage for storing containers; and a plurality of transportation robotsconfigured to autonomously transport containers from said storage tosaid picking station.
 13. (canceled)
 14. A method comprising: obtainingan instruction to pick an item from a container; first determining thatan operation plan for a robotic picking apparatus cannot be devisedautonomously; in response to said first determining, second determiningthat the operation plan cannot be devised based on additional humanprovided input; and in response to said second determining, receivinguser instructions for remotely controlling the robotic pickingapparatus, whereby the robotic picking apparatus is operated in remoteoperation mode and controlled by a user; and moving the robotic pickingapparatus based on the user instructions.
 15. (canceled)
 16. The methodof claim 14, wherein in response to said second determining, selecting auser out of a plurality of users to control the robotic pickingapparatus, wherein said selecting is performed from a plurality of humanusers located remotely from the robotic picking apparatus.
 17. Themethod of claim 16, wherein a ratio between a total number of humanusers that are tasked with assisting or controlling robotic pickingapparatuses and a total number of the robotic picking apparatuses is atleast 1:100.
 18. The method of claim 14, wherein said first determiningcomprises; calculating a first confidence measurement of the capabilityof autonomous determination of the operation plan, and comparing thefirst confidence measurement with a first threshold; and wherein saidsecond determining comprises: calculating a second confidencemeasurement of the capability of determination of the operation planusing additional human provided input, and comparing the secondconfidence measurement with a second threshold.
 19. The method of claim18, wherein at least one of the first threshold and the second thresholdis modified based on availability of human users to provide input. 20.The method of claim 18, wherein the first threshold is modified based onavailability of manned stations for remote assistance; and wherein thesecond threshold is modified based on availability of manned stationsfor remote control.
 21. The method of claim 14 further comprisingobtaining an order to fulfill; and fulfilling the order by instructingthe robotic picking apparatus to place items in a shipping container inaccordance with the order, wherein said fulfilling comprises: therobotic picking apparatus picking and placing at least one item in theshipping container in an autonomous mode and without user input; and therobotic picking apparatus picking and placing at least one item in ahuman-assisted mode.
 22. The method of claim 14 further comprisingobtaining an order to fulfill; and fulfilling the order by instructingthe robotic picking apparatus to place items in a shipping container inaccordance with the order, wherein said fulfilling comprises: therobotic picking apparatus picking and placing at least one item in theshipping container in a remote-controlled mode; and the robotic pickingapparatus picking and placing at least one item in the shippingcontainer in an autonomous mode and without user input.
 23. A methodcomprising: obtaining a instruction to pick an item from a container;computing an estimated cost of operating a robotic picking apparatus inan autonomous mode, wherein the estimated cost of operating the roboticpicking apparatus in the autonomous mode is based on a probability ofsuccess of operating the robotic picking apparatus in the autonomousmode and estimated cost of damage to the item; computing an estimatedcost of operating the robotic picking apparatus in a human-assistedmode, wherein the estimated cost of operating the robotic pickingapparatus in the human-assisted mode is based on a probability ofsuccess of operating the robotic picking apparatus in the human-assistedmode, the estimated cost of damage to the item and a cost of labor inhuman-assisted model; computing an estimated cost of operating therobotic picking apparatus in a remote-controlled mode, wherein theestimated cost of operating the robotic picking apparatus in theremote-controlled mode is based on a probability of success of operatingthe robotic picking apparatus in the remote-controlled mode, theestimated cost of damage to the item and a cost of labor inremote-controlled mode; selecting between the autonomous mode, thehuman-assisted mode and the remote-controlled mode based on theestimated costs; and operating the robotic picking apparatus to pick upthe item from the container in the selected mode.
 24. A methodcomprising: obtaining an order of a customer to fulfill, wherein theorder comprises ordered items; instructing one or more transportationrobots to transport containers to a picking station comprising a roboticpicking apparatus, wherein the containers comprise the ordered items;picking, using the robotic picking apparatus, the ordered items andplacing the ordered items in a shipping container, wherein said pickingcomprises: picking at least one ordered item in an autonomous mode;picking one or more ordered items that cannot be picked in theautonomous mode, wherein said picking the one or more items comprises:for each item of the one or more ordered items, determining whether toutilize a human-assisted mode or a remote-controlled mode for pickingthe each item; obtaining human input based on the determined mode; andpicking the each item in the determined mode using the human input; andtransporting the shipping container to the customer; whereby the orderis fulfilled by a robotic warehouse with limited human assistance.